A Syntactic Justification for Occam’s razor

Informally, Occam’s razor states, “Given two hypotheses which equally agree with the observed data, choose the simpler”, and has become a central guiding heuristic in the empirical sciences and in particular machine learning. We criticize previous arguments for the validity of Occam’s razor. The nature of hypotheses spaces is explored and we observe a correlation between the complexity of a concept yielded by a hypothesis and the frequency with which it is represented when the hypothesis space is uniformly sampled. We argue that there is not a single best hypothesis but a set of hypotheses which give rise to the same predictions (i.e. the hypotheses are semantically equivalent), whereas Occam’s razor suggests there is a single best hypothesis. We prefer one set of hypotheses over another set because it is the larger set (and therefore the most probable) and the larger set happens to contain the simplest consistent hypothesis. This gives the appearance that simpler hypotheses generalize better. Thus, the contribution of this paper is the justification of Occam’s razor by a simple counting argument.